{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2020 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/GoogleCloudPlatform/keras-idiomatic-programmer/blob/master/books/deep-learning-design-patterns/Workshops/Junior/Deep%20Learning%20Design%20Patterns%20-%20Workshop%20-%20Chapter%203.ipynb\">\n",
    "<img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Learning Design Patterns - Code Labs\n",
    "\n",
    "## Lab Exercise #7 - Get Familiar with Alternative Connectivity Patterns\n",
    "\n",
    "## Prerequistes:\n",
    "\n",
    "    1. Familiar with Python\n",
    "    2. Completed Chapter 3: Alternative Connectivity Patterns\n",
    "\n",
    "## Objectives:\n",
    "\n",
    "    1. Code a Squeeze-Excitation (SE) Link\n",
    "    2. Add the SE-Link to an Inception V1 block.\n",
    "    3. Construct a mini SE-Inception V1 model\n",
    "    4. Add the SE-Link to a DenseNet block.\n",
    "    5. Construct a mini SE-DenseNet model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code a SE Link\n",
    "\n",
    "Let's now code a squeeze and excitation (SE) link. Remember, uses dense layers with non-linear activation functions.\n",
    "\n",
    "<img src='https://github.com/GoogleCloudPlatform/keras-idiomatic-programmer/blob/master/books/deep-learning-design-patterns/Workshops/Junior/se-link.jpg?raw=true'>\n",
    "\n",
    "You will need to:\n",
    "\n",
    "    1. Determine the number of input channels to the (se) link.\n",
    "    2. Set the number of filters for the squeeze.\n",
    "    3. multiply the output of the excitation with the input to the link.\n",
    "    4. Set the ratio for the squeeze."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Input, Model\n",
    "from tensorflow.keras.layers import Conv2D, ReLU, BatchNormalization, Concatenate, Dense\n",
    "from tensorflow.keras.layers import GlobalAveragePooling2D, Reshape, Multiply\n",
    "\n",
    "def se_link(inputs, **params): \n",
    "    ratio = params['ratio']\n",
    "    \n",
    "    # Get the number of input channels (feature maps)\n",
    "    # HINT: It is always the last dimension\n",
    "    n_filters = inputs.shape[??]\n",
    "    \n",
    "    # Global Average Pooling to 1x1xC\n",
    "    outputs = GlobalAveragePooling2D()(inputs)\n",
    "    outputs = Reshape((1, 1, n_filters))(outputs)\n",
    "    \n",
    "    # Dense w/ReLU to 1x1xC/ratio\n",
    "    # HINT: divide the input number of filters by the ratio, and use the same activation as you do with Conv2D.\n",
    "    outputs = Dense(n_filters // ??, activation='??')(outputs)\n",
    "    \n",
    "    # Dense w/Sigmoid to 1x1xC\n",
    "    outputs = Dense(n_filters, activation='sigmoid')(outputs)\n",
    "    \n",
    "    # Multiply the SE output with the inputs\n",
    "    # HINT: The output from the dense layer with the sigmoid activation\n",
    "    outputs = Multiply()([inputs, ??])\n",
    "    \n",
    "    return outputs\n",
    "\n",
    "inputs = Input((32, 32, 3))\n",
    "# Set the squeeze ratio to 4\n",
    "# HINT: It's the number 4.\n",
    "outputs = se_link(inputs, **{'ratio': ??})\n",
    "model = Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the module using summary method\n",
    "\n",
    "It should look like below.\n",
    "\n",
    "```\n",
    "Model: \"model_6\"\n",
    "__________________________________________________________________________________________________\n",
    "Layer (type)                    Output Shape         Param #     Connected to                     \n",
    "==================================================================================================\n",
    "input_19 (InputLayer)           [(None, 32, 32, 3)]  0                                            \n",
    "__________________________________________________________________________________________________\n",
    "global_average_pooling2d_16 (Gl (None, 3)            0           input_19[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "reshape_2 (Reshape)             (None, 1, 1, 3)      0           global_average_pooling2d_16[0][0]\n",
    "__________________________________________________________________________________________________\n",
    "dense_22 (Dense)                (None, 1, 1, 0)      0           reshape_2[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "dense_23 (Dense)                (None, 1, 1, 3)      3           dense_22[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "multiply_4 (Multiply)           (None, 32, 32, 3)    0           input_19[0][0]                   \n",
    "                                                                 dense_23[0][0]                   \n",
    "==================================================================================================\n",
    "Total params: 3\n",
    "Trainable params: 3\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement a mini Inception V1 with SE-Link\n",
    "\n",
    "Let's construct a mini-inception V1 model with a SE-Link. We will call it a SE-Inception V1. Our model will consist of:\n",
    "\n",
    "    1. single se-inception block\n",
    "    2. classifier\n",
    "    \n",
    "You will need to:\n",
    "\n",
    "    1. Complete the parameter arguments to the se_link() call.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def se_inception_block(inputs, **params):\n",
    "    # pooling branch\n",
    "    x1 = MaxPooling2D((3, 3), strides=(1, 1), padding='same')(inputs)\n",
    "    x1 = Conv2D(64, (1, 1), strides=(1, 1), padding='same')(x1)\n",
    "    \n",
    "    # 1x1 branch\n",
    "    x2 = Conv2D(64, (1, 1), strides=(1, 1), padding='same', activation='relu')(inputs)\n",
    "    \n",
    "    # 3x3 branch\n",
    "    x3 = Conv2D(64, (1, 1), strides=(1, 1), padding='same')(inputs)\n",
    "    x3 = Conv2D(96, (3, 3), strides=(1, 1), padding='same', activation='relu')(x3)\n",
    "    \n",
    "    # 5x5 branch\n",
    "    x4 = Conv2D(64, (1, 1), strides=(1, 1), padding='same')(inputs)\n",
    "    x4 = Conv2D(48, (5, 5), strides=(1, 1), padding='same', activation='relu')(x4)\n",
    "    \n",
    "    outputs = Concatenate()([x1, x2, x3, x4])\n",
    "    \n",
    "    # Insert the SE-Link \n",
    "    # HINT: the input to the SE-Link is the output from the concatenation. Use **kwargs syntax to pass down the ratio\n",
    "    outputs = se_link(??)\n",
    "    return outputs\n",
    "\n",
    "def classifier(inputs, n_classes):\n",
    "    outputs = GlobalAveragePooling2D()(inputs)\n",
    "    outputs = Dense(n_classes, activation='softmax')(outputs)\n",
    "    return outputs\n",
    "    \n",
    "inputs = Input((32, 32, 3))\n",
    "outputs = se_inception_block(inputs, **{'ratio': 4})\n",
    "outputs = classifier(outputs, 10)\n",
    "model = Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the module using summary method\n",
    "\n",
    "It should look like below.\n",
    "\n",
    "```\n",
    "Layer (type)                    Output Shape         Param #     Connected to                     \n",
    "==================================================================================================\n",
    "input_10 (InputLayer)           [(None, 32, 32, 3)]  0                                            \n",
    "__________________________________________________________________________________________________\n",
    "max_pooling2d_3 (MaxPooling2D)  (None, 32, 32, 3)    0           input_10[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_20 (Conv2D)              (None, 32, 32, 64)   256         input_10[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_22 (Conv2D)              (None, 32, 32, 64)   256         input_10[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_18 (Conv2D)              (None, 32, 32, 64)   256         max_pooling2d_3[0][0]            \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_19 (Conv2D)              (None, 32, 32, 64)   256         input_10[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_21 (Conv2D)              (None, 32, 32, 96)   55392       conv2d_20[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_23 (Conv2D)              (None, 32, 32, 48)   76848       conv2d_22[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "concatenate_2 (Concatenate)     (None, 32, 32, 272)  0           conv2d_18[0][0]                  \n",
    "                                                                 conv2d_19[0][0]                  \n",
    "                                                                 conv2d_21[0][0]                  \n",
    "                                                                 conv2d_23[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "global_average_pooling2d_5 (Glo (None, 272)          0           concatenate_2[0][0]              \n",
    "__________________________________________________________________________________________________\n",
    "dense_6 (Dense)                 (None, 4)            1092        global_average_pooling2d_5[0][0] \n",
    "__________________________________________________________________________________________________\n",
    "dense_7 (Dense)                 (None, 16)           80          dense_6[0][0]                    \n",
    "==================================================================================================\n",
    "Total params: 134,436\n",
    "Trainable params: 134,436\n",
    "Non-trainable params: 0\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Let's do a bit of training with your mini-inception V3 + SE-Link (SE-InceptionV1)\n",
    "\n",
    "### Dataset\n",
    "\n",
    "Let's get the tf.Keras builtin dataset for CIFAR-10. These are 32x32 color images (3 channels) of 10 classes (airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks). We will preprocess the image data (not covered yet)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import cifar10\n",
    "import numpy as np\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "x_train = (x_train / 255.0).astype(np.float32)\n",
    "x_test  = (x_test / 255.0).astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "Let's train the model  for 3 epochs.\n",
    "\n",
    "Because it just a few epochs, you test accuracy may vary from run to run. For me, it was 40.5%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "model.fit(x_train, y_train, epochs=3, batch_size=32, validation_split=0.1, verbose=1)\n",
    "model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add another SE-Inception V1 block\n",
    "\n",
    "Let's improve our model by adding a second SE-InceptionV1 block, but we will change the compression ratio in the SE-Link.\n",
    "\n",
    "You will need to:\n",
    "\n",
    "    1. Complete the parameter arguments for the second call to the se_inception_block() method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = Input((32, 32, 3))\n",
    "outputs = se_inception_block(inputs, **{'ratio': 4})\n",
    "# Add the second se-inception block with ratio = 2\n",
    "# HINT: use key-value syntax\n",
    "outputs = se_inception_block(outputs, **{??})\n",
    "outputs = classifier(outputs, 10)\n",
    "model = Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "Let's train the model for just 2 epochs.\n",
    "\n",
    "By going deeper (adding an inception block with se-link), we get higher accuracy in 2 epochs vs. 3 epochs in the more shallower version. For me, it was 51.6%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "model.fit(x_train, y_train, epochs=2, batch_size=32, validation_split=0.1, verbose=1)\n",
    "model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement a mini-DenseNet with SE-Link\n",
    "\n",
    "Let's construct a mini-densenet model with a SE-Link. We will call it a SE-DenseNet. Our model will consist of:\n",
    "\n",
    "    1. se-densenet group with two blocks\n",
    "    2. classifier\n",
    "    \n",
    "You will need to:\n",
    "\n",
    "    1. Extract the block and ratio settings.\n",
    "    2. Set the linear projection convolution to be a 1x1 kernel\n",
    "    3. Pass the block-level parameters to the se_dense_block() calls.\n",
    "    4. Pass the group level parameter ratio to the group() call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def group(inputs, **params):\n",
    "    # Get the blocks and ratio data\n",
    "    # HINT: they are keys in params\n",
    "    blocks = ??\n",
    "    ratio  = ??\n",
    "    \n",
    "    outputs = inputs\n",
    "    for block_params in blocks:\n",
    "        # Construct the next SE-DenseNet\n",
    "        # HINT: pass the parameters for this block as kwargs\n",
    "        outputs = se_dense_block(outputs, **??, ratio=ratio)\n",
    "    return outputs\n",
    "        \n",
    "def se_dense_block(inputs, **params):\n",
    "    n_filters = params['n_filters']\n",
    "    \n",
    "    # 1x1 linear projection (dimensionality expansion)\n",
    "    outputs = BatchNormalization()(inputs)\n",
    "    outputs = ReLU()(outputs)\n",
    "    # HINT: its a 1x1 kernel\n",
    "    outputs = Conv2D(n_filters * 4, ??, strides=(1, 1), padding='same')(outputs)\n",
    "    \n",
    "    # Bottleneck (dimensionality restoration)\n",
    "    outputs = BatchNormalization()(outputs)\n",
    "    outputs = ReLU()(outputs)\n",
    "    outputs = Conv2D(n_filters, (3, 3), strides=(1, 1), padding='same')(outputs)\n",
    "    \n",
    "    # Add the SE-Link before the concatentation of the inputs to the outputs\n",
    "    # HINT: pass thru the kwargs that were passed into the block method\n",
    "    outputs = se_link(outputs, **??)\n",
    "    \n",
    "    # feature reuse\n",
    "    outputs = Concatenate()([inputs, outputs])\n",
    "    return outputs\n",
    "\n",
    "inputs = Input((32, 32, 3))\n",
    "# Set the squeeze ratio to 4\n",
    "# HINT: pass it as kwargs ratio and value 4 -- that is a key/value pair\n",
    "outputs = group(inputs, **{ 'blocks': [{'n_filters': 32}, {'n_filters': 64}], ?? })\n",
    "outputs = classifier(outputs, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the module using summary method\n",
    "\n",
    "It should look like below.\n",
    "\n",
    "```\n",
    "Layer (type)                    Output Shape         Param #     Connected to                     \n",
    "==================================================================================================\n",
    "input_29 (InputLayer)           [(None, 32, 32, 3)]  0                                            \n",
    "__________________________________________________________________________________________________\n",
    "batch_normalization_16 (BatchNo (None, 32, 32, 3)    12          input_29[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "re_lu_16 (ReLU)                 (None, 32, 32, 3)    0           batch_normalization_16[0][0]     \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_88 (Conv2D)              (None, 32, 32, 128)  512         re_lu_16[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "batch_normalization_17 (BatchNo (None, 32, 32, 128)  512         conv2d_88[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "re_lu_17 (ReLU)                 (None, 32, 32, 128)  0           batch_normalization_17[0][0]     \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_89 (Conv2D)              (None, 32, 32, 32)   36896       re_lu_17[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "global_average_pooling2d_28 (Gl (None, 32)           0           conv2d_89[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "reshape_8 (Reshape)             (None, 1, 1, 32)     0           global_average_pooling2d_28[0][0]\n",
    "__________________________________________________________________________________________________\n",
    "dense_39 (Dense)                (None, 1, 1, 8)      264         reshape_8[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "dense_40 (Dense)                (None, 1, 1, 32)     288         dense_39[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "multiply_10 (Multiply)          (None, 32, 32, 32)   0           conv2d_89[0][0]                  \n",
    "                                                                 dense_40[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "concatenate_17 (Concatenate)    (None, 32, 32, 35)   0           input_29[0][0]                   \n",
    "                                                                 multiply_10[0][0]                \n",
    "__________________________________________________________________________________________________\n",
    "batch_normalization_18 (BatchNo (None, 32, 32, 35)   140         concatenate_17[0][0]             \n",
    "__________________________________________________________________________________________________\n",
    "re_lu_18 (ReLU)                 (None, 32, 32, 35)   0           batch_normalization_18[0][0]     \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_90 (Conv2D)              (None, 32, 32, 256)  9216        re_lu_18[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "batch_normalization_19 (BatchNo (None, 32, 32, 256)  1024        conv2d_90[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "re_lu_19 (ReLU)                 (None, 32, 32, 256)  0           batch_normalization_19[0][0]     \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_91 (Conv2D)              (None, 32, 32, 64)   147520      re_lu_19[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "global_average_pooling2d_29 (Gl (None, 64)           0           conv2d_91[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "reshape_9 (Reshape)             (None, 1, 1, 64)     0           global_average_pooling2d_29[0][0]\n",
    "__________________________________________________________________________________________________\n",
    "dense_41 (Dense)                (None, 1, 1, 16)     1040        reshape_9[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "dense_42 (Dense)                (None, 1, 1, 64)     1088        dense_41[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "multiply_11 (Multiply)          (None, 32, 32, 64)   0           conv2d_91[0][0]                  \n",
    "                                                                 dense_42[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "concatenate_18 (Concatenate)    (None, 32, 32, 99)   0           concatenate_17[0][0]             \n",
    "                                                                 multiply_11[0][0]                \n",
    "__________________________________________________________________________________________________\n",
    "global_average_pooling2d_30 (Gl (None, 99)           0           concatenate_18[0][0]             \n",
    "__________________________________________________________________________________________________\n",
    "dense_43 (Dense)                (None, 10)           1000        global_average_pooling2d_30[0][0]\n",
    "==================================================================================================\n",
    "Total params: 199,512\n",
    "Trainable params: 198,668\n",
    "Non-trainable params: 844\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "Let's train the model for just 2 epochs.\n",
    "\n",
    "For me, it was 42.2%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "model.fit(x_train, y_train, epochs=2, batch_size=32, validation_split=0.1, verbose=1)\n",
    "model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of Lab Exercise"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
